Phonological memory, attention control, and musical ability: Effects of individual differences on rater judgments of second language speech

This study examines how listener judgments of second language speech relate to individual differences in listeners’ phonological memory, attention control, and musical ability. Sixty native English listeners (30 music majors, 30 nonmusic majors) rated 40 nonnative speech samples for accentedness, comprehensibility, and fluency. The listeners were also assessed for phonological memory (serial recognition), attention control (trail making), and musical aptitude. Results showed that music majors assigned significantly lower scores than nonmusic majors solely for accentedness, particularly for low ability second language speakers. However, the ratings were not significantly affected by individual differences in listeners’ phonological memory and attention control, which implies that these factors do not bias listeners’ subjective judgments of speech. Implications for psycholinguistic research and for high-stakes speaking assessments are discussed. As universities and other postsecondary institutions seek to attract an increasingly diverse student body, they face the responsibility of providing valid assessments of incoming students’ language ability, especially when the students’ mother tongue is not the language of instruction (Cheng, Myles, & Curtis, 2004). There have been attempts to develop technology-based, automated assessment instruments for spoken English (Downey, Farhady, Present-Thomas, Suzuki, & Van Moere, 2008). However, the most commonly used second language (L2) speaking tests in academic settings, whether they rely on recorded speaking prompts (e.g., Test © Cambridge University Press 2010 0142-7164/10 $15.00 Applied Psycholinguistics 32:1 114 Isaacs & Trofimovich: Cognitive factors and ratings of L2 pronunciation of English as a Foreign Language [TOEFL] Internet-Based Test, Test of Spoken English [TSE]) or on face-to-face interaction (e.g., International English Language Testing System [IELTS]), are scored by human raters (Templer, 2004). Rater judgments in academic settings are central to high stakes decisions, including whether a candidate is admitted to the university, placed in a remedial language course, or awarded a teaching assistantship. Although rater judgments are often used as the chief source of evidence of L2 speakers’ language proficiency in academic settings, such judgments might not always be reliable (e.g., when scoring is not internally consistent) or valid (e.g., when scoring is influenced by factors extraneous to the construct being measured). That is, raters’ judgments might reflect not simply speakers’ performance, but also individual differences among raters themselves. For example, ongoing validation research of standardized L2 tests such as the TOEFL, TSE, and IELTS has revealed various sources of rater variability (Brown, Iwashita, & McNamara, 2005; Myford & Wolfe, 2000; Taylor, 2007), including raters’ experience (Cumming, 1990), gender (O’Loughlin, 2007), the relative weight they place on different scoring criteria (Eckes, 2008), and their native (first) language (L1) background (Kim, 2009). What is not known, however, is whether other sources of rater variability, for example, those related to individual differences in raters’ cognitive abilities (e.g., phonological memory, attention control, or music aptitude), also influence raters’ assessments of spoken language. In the present study, we therefore investigated whether individual differences in raters’ phonological memory (auditory working memory capacity), attention control (ability to allocate attention efficiently), or musical skill (musical aptitude) influence raters’ judgments of L2 speech on dimensions of accentedness, comprehensibility, and fluency. Accentedness is defined here as listeners’ judgments of how closely the pronunciation of an utterance approaches that of a native speaker (Munro & Derwing, 1999). Comprehensibility refers to listeners’ perceptions of how easily they understand an utterance (Munro & Derwing, 1999). Fluency denotes listeners’ assessments of how smoothly and rapidly an utterance is spoken (cf. Derwing, Rossiter, Munro, & Thomson, 2004). Our overall goal was to determine how phonological memory, attention control, and musical ability could contribute to listeners’ perceptual judgments of L2 speech and, as a result, could influence their scoring decisions.